Identification and ranking of news stories of interest

ABSTRACT

Methods, systems, and apparatus, including computer program products, for ranking news articles. A plurality of news articles referenced in one or more hub pages are identified, where each of the hub pages include respective references to one or more of the news articles. A score component is derived for a news article from a measure of a prominence of the news article in the hub pages that includes a reference to the news article.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation application of, and claims priorityto, U.S. patent application Ser. No. 11/938,705, entitled IDENTIFICATIONAND RANKING OF NEWS STORIES OF INTEREST, to inventor Sharad Jain, whichwas filed on Nov. 12, 2007. The disclosure of the foregoing applicationis incorporated herein by reference in its entirety.

BACKGROUND

This specification relates generally to search engines.

The World Wide Web (the “web”) contains a vast amount of information. Auser can use a search engine to find the information on the web in whichthe user is interested. For example, a user can use the search engine tofind news articles. The search engine retrieves news articles from websites and presents them to the user as search results, which generallyeach include some text describing the corresponding news article and alink to the article itself. The search engine can group multiplearticles that are reporting on the same news story together and presentthe multiple articles as a group. When deciding the order ofpresentation for new articles and groups of news articles, the searchengine can consider several factors, such as the date of publication,the quality of the source, and so on.

SUMMARY

In general, one aspect of the subject matter described in thisspecification can be embodied in methods that include the actions ofidentifying a plurality of news articles referenced in one or more hubpages, where each of the hub pages include respective references to oneor more of the plurality of news articles; and deriving a scorecomponent for each of the news articles from a measure of a prominenceof the respective news article in each of the hub pages that includes areference to the respective news article. Other embodiments of thisaspect include corresponding systems, apparatus, computer programproducts, and computer readable media.

In general, another aspect of the subject matter described in thisspecification can be embodied in methods that include the actions ofidentifying a news article and one or more hub pages, where each hubpage include a reference to the news article; and determining an articlescore for the news article from a hub-page specific score for the newsarticle for each of the hub pages, where the hub-page specific score foreach hub page is determined based on a relative prominence of thereference to the news article on the hub page relative to otherreferences to other content on the hub page. Other embodiments of thisaspect include corresponding systems, apparatus, computer programproducts, and computer readable media.

Particular embodiments of the subject matter described in thisspecification can be implemented to realize one or more of the followingadvantages. A news search engine can tap into the knowledge embodied inthe editorial decision-making involved in the positioning of hyperlinksto news articles to determine the importance of news articles.Importance of news articles or groups of news articles can bedetermined. News can be broken faster by lessening the influence of thesize of a group of news articles on the timing with respect to when newsis broken.

The details of one or more embodiments of the subject matter describedin this specification are set forth in the accompanying drawings and thedescription below. Other features, aspects, and advantages of thesubject matter will become apparent from the description, the drawings,and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an example search environment.

FIG. 2 illustrates an example system architecture for a searchenvironment.

FIG. 3 is a flow diagram illustrating an example process for determiningpositions in hub pages of references to news articles and scoring thenews articles based on the determined positions.

FIGS. 4A and 4B illustrate example hub pages.

FIG. 5A illustrates another example hub page.

FIG. 5B illustrates a Document Object Model graph of the hub pageillustrated in FIG. 5A.

Like reference numbers and designations in the various drawings indicatelike elements.

DETAILED DESCRIPTION

FIG. 1 illustrates an example search environment 100. The searchenvironment 100 includes a client device 102, a search system 104, andone or more content hosts 106. The client device 102, search system 104,and content hosts 106 can communicate over one or more networks 108. Insome implementations, content hosts 106 include servers that store andserve resources. Examples of networks 108 include local area networks(LANs), wide area networks (WANs), wireless (e.g., Wi-Fi) networks,mobile phone networks, and the Internet.

The search system 104 indexes resources hosted by content hosts 106. Thesearch system 104 crawls the content hosts 106 to identify resourcesavailable for indexing. In some implementations, the search system 104also stores copies of the indexed resources. Examples of resourcesinclude documents (e.g., web pages, Portable Document Format (PDF)documents, text files, or word processing documents), audio files, videofiles, images, and so on. In some implementations, the identifiedresources include news articles, which are documents that includetextual content describing or reporting news events.

The search system 104 can receive search queries for resources (e.g.,from a client device 102). For a respective search query, the searchsystem 104 searches the index for indexed resources that satisfy thesearch query. A query includes one or more terms (e.g., words, phrases,characters, ideograms, or numbers). The search system 104 generatessearch results that identify resources that satisfy the search query. Insome implementations, the search system 104 scores and ranks the searchresults. The search results are transmitted from the search system 104to a client device 102 for presentation to a user.

The client device 102 can be any device that can communicate with thesearch system 104 and the content hosts 106 through the one or morenetworks 108. In some implementations, the client device 102 includes aclient application (e.g., a web browser) that can access the searchsystem 104 and/or the content hosts 106. Examples of client devices 102include desktop computers, notebook computers, tablet computers,personal digital assistants (PDAs), mobile phones, smartphones, mediaplayers, game consoles, portable gaming devices, set-top boxes, and soon.

In some implementations, the search system 104 includes a user interfacethat is accessible through the client device 102. The user interface caninclude a front page. The front page can display a search field, where auser can enter a search query for resources (e.g., news articles). Thefront page can also display references to one or more news articles. Thereferences that are displayed on the front page can be selected based onprominence measures of the news articles corresponding to thereferences. Prominence measures of news articles are further describedbelow.

FIG. 2 illustrates an example system architecture 200 for a searchenvironment. The system architecture 200 includes a search engine 206and a client device 102. The search engine 206 and the client device 102can communicate over one or more networks 108. In some implementations,search engine 206 is an implementation of search system 104.

In some implementations, the search engine 206 includes a search engineserver 208, crawling module 210, article reference location module 212,article scoring module 214, article grouping module 216, and articlegroup scoring module 218. In other implementations, the functionality ofthe search engine 206 is organized in other ways. The search engine 206also includes a sources repository 220 and an articles repository 222.

The search engine server 208 receives queries from user devices 202,generates the responses to the queries (which generally include searchresults), and transmits the responses to the user devices 202.Generally, the search results include links to documents (e.g., onlinenews articles).

The crawling module 210 crawls content hosts (e.g., content hosts 106)for resources to index. The crawling module 210 can crawl for newsarticles. In some implementations, the crawling module 210 crawls fornews articles in content hosts that are identified in the sourcesrepository 220 as sources of news articles. For example, the sourcesrepository 220 includes information on hub pages that reference newsarticles. The crawling module 210 crawls these hub pages for referencesto news article documents and follows the references to crawl the newsarticles. In some implementations, the crawling module 210 alsodetermines whether crawled pages should be identified as hub pages, andproviding information identifying hub pages to the sources repository220.

The article reference location module 212 determines the positions, inhub pages, of references to news articles. The article referencelocation module 212 analyzes the hub pages that include references tonews articles crawled by the crawling module 210 to determine thepositions of the references within the hub pages.

Article scoring module 214 determines scores for news articles. In someimplementations, the article scoring module 214 determines scores fornews articles based on, among other criteria, the positions ofreferences to the news articles in hub pages, as further describedbelow.

The article grouping module 216 groups news articles into groups. A newsarticle group includes news articles that are determined to describe,report on, or be related to a particular news story or event. Thearticle grouping module 216 groups news articles into the news articlegroups based on an analysis of the content of the news articles.

The article group scoring module 218 scores the news article groupsbased on the scores of the articles within the groups and, in someimplementations, based on other criteria as well. In someimplementations, the article group scoring module 218 calculates a scorefor a news article group from the scores of the news articles in thegroup, a rate of publication of news articles in the group within a timeperiod, and optionally other criteria (e.g., the positions of thereferences to news articles in hub pages).

In some implementations, a sources repository 220 includes a database ofsources of news articles and hub pages associated with these sources.The term “source” is used in this specification to refer to an entitythat publishes or makes available news articles on the web. The sourcesrepository 220 can be populated manually or by a crawler that finds newsarticle sources and hub pages according to predefined rules or criteria.For example, a crawler can find pages that are identified as news pagesin their page metadata or titles. In some implementations, a newsarticle source has multiple hub pages. For example, a general newssource (e.g., CNN.com) can have an overall hub page (e.g., the CNN.comhomepage) and subsidiary hub pages (e.g., homepages for a businesssection, a sports section, an entertainment section, and so on).

In some implementations, the sources repository 220 also stores qualitymetrics or ratings for the new sources.

The articles repository 222 stores an index of news article crawled bythe crawling module 210 and the news article groups. The articlesrepository 222 can also store other information (e.g., metadata) relatedto the news articles and news article groups. In some implementations,the articles repository 222 also includes copies (e.g., cache copies) ofat least some of the crawled articles.

The client device 102 can include an application 204. In someimplementations, the application 204 is a web browser. A user can usethe application 204 to access the search engine server 208, to submitqueries, and to receive search results.

FIG. 3 is a flow diagram illustrating an example process 300 fordetermining positions in hub pages of references to news articles andscoring the news articles based on the determined positions. Forconvenience, the process 300 will be described in reference to a system(e.g., system 200) that performs the process.

The system 200 begins with a set of hub pages (301). The term “hub page”is used in this specification to refer to a web page that includes aplurality of references to news articles, where the number of referencesto news articles in the web page is above a predetermined threshold andwhere the references to the news articles are not necessarily sorted bytime. The references can link to news articles in the same domain as thehub page or to news articles in a different domain. A reference in thehub page to a news article includes a hyperlink to a corresponding newsarticle and optionally content related to the news article (e.g., asnippet, subtitle, or a synopsis of the news article, an image relatedto the news article).

In some implementations, the crawling module 210 crawls one or more hubpages that are specified in the sources repository 220. The sourcesrepository 220 can be populated with hub pages manually (e.g., by anadministrator navigating to a web page, deciding that the web pagesatisfies one or more criteria for being a hub page, and adding it tothe sources repository 220) or automatically. For example, the crawlingmodule 210 can crawl web sites of known news sources to find hub pages.In some implementations, the hub pages are automatically or manuallyidentified based on particular criteria. Web pages having more than athreshold number of outgoing news article hyperlinks can be identifiedas hub pages, for example. A hyperlink is identified as a hyperlink to anews article if the destination document of the hyperlink satisfies oneor more criteria, e.g., an amount of text in the destination documentabove some minimum, the text formatted in a particular way, etc.

In some implementations, the sources repository 220 includes indicationsof quality for the sources included in the repository 220. For example,one or more of the sources in the sources repository 220 can be markedas a high-quality source. As another example, each of the sources in thesources repository 220 can have a quality rating. The markings of highquality or the quality ratings can be added to the repository based on amanual review of the sources and associated hub pages. In someimplementations, the crawling module 210 crawls only the hub pages ofsources marked as high-quality or crawls the hub pages of sources whoserating is above a predetermined threshold.

In some implementations, hub pages are identified using a combination ofan automated process and a manual process. The automated processinvolves a crawler (e.g., crawling module 210) crawling web pages andidentifying from the crawled web pages candidate hub pages. Web pageshaving more than a threshold number of outgoing news article hyperlinkscan be identified as candidate hub pages, for example. The candidate hubpages are then reviewed by a person as a second level of verificationthat the candidate hub page does qualify as a hub page. Candidate hubpages that are verified are added to the sources repository 220. In someimplementations, the human reviewer also makes a judgment regarding thequality of the candidate hub page. Those candidate hub pages that arejudged to be of acceptable quality are added to the sources repository220.

The system crawls hub pages and identifies news articles referenced inthe hub pages (302). In some implementations, the crawling module 210crawls the identified hub pages and, when crawling a hub page,identifies references as references to news articles in the hub pageusing one or more rules. In some implementations, the crawling module210 identifies references to news articles by following hyperlinks inthe hub page and applying the rules to the target page of the hyperlink.An example of a rule is related to the length of text that is betweenHTML “<DIV>” and “</DIV>” tags in the resulting page. If the longestlength of text between <DIV>tags in the resulting page is longer than aminimum length, for example, and any other rules are satisfied, then theresulting page is identified as a news article, and the correspondinghyperlink (and optionally any associated snippet and other informationrelated to the identified news article) is identified as a reference tothe news article. Another example rule is whether, in the target page,the number of pairs of “<DIV>” and “</DIV>” tags that have at least someminimum amount of text between the respective DIV tags is greater thansome threshold. If the number of DIV tag pairs with at least the minimumamount of text is greater than the threshold, the target page is morelikely to be identified as a news article. Other examples of rules foridentifying news articles include whether, in the target page: thenumber of text paragraphs set off by a leading <P>tag or a <P></P>tagpair that includes at least a minimum amount of text exceeds somethreshold; whether the number of text blocks set off as table rows(e.g., with <tr></tr>tag pairs) that includes at least a minimum amountof text exceeds some threshold; and whether a number of URL's in thetarget page text is below some threshold with respect to the target pagetext.

The system determines the positions of the news article references inthe hub pages (304). In some implementations, the article referencelocation module 212 analyzes the hub page to determine the positions ofthe references in the hub page as it would appear when the hub page isrendered for display (e.g., when rendered by a web browser). In someimplementations, this analysis includes analyzing the HTML code, theDocument Object Model (DOM), and/or a style sheet (e.g., Cascading StyleSheet (CSS)) of a hub page. By analyzing the HTML code, DOM, and/or CSSof the hub page, a rendered layout of the hub page is determined, andthe positions of the references are determined based on the renderedlayout.

Different hub pages have different rendered layouts. In some hub pages,the news article references are arranged in a vertical list, from top tobottom, when the hub page is rendered. An example is the example hubpage illustrated in FIG. 4A. In some implementations, the positions ofnews article references in the vertical list are determined by parsingthe HTML source code of the hub page from top to bottom and identifyingthe hyperlink URLs and their positions relative to each other. A newsarticle can be assigned a reference position score based on the positionof the corresponding news article reference in the vertical list andoptionally the total number of news article references in the hub page.For example, if there are 10 references, the topmost reference can beassigned a reference position score of 10 (highest score for being thetopmost reference of 10 references) or a normalized reference positionscore of 1 (10 for being the topmost reference of 10 references, dividedby the number of references). As another example, the reference positionscore can be calculated using a formula where the position is an input.An example of a formula for calculating a reference position score isreference_position_score=C1/power(reference_order, C2), where C1 and C2are positive constants, reference_order is a value assigned to areference based on the determined position (e.g., 1 for the top-most orhighest ordered reference, 2 for the next reference in the order, and soon), and power(reference_order, C2) is reference_order to the power C2.In some implementations, values for C1 and C2 are 10 and 0.5,respectively. In some other implementations, other values for C1 and C2are possible. Other ways to assign a reference position score arepossible.

In some other hub pages, the hub page as rendered and displayed includesnews article references that are arranged vertically and horizontally.An example is the example hub page illustrated in FIG. 4B. For these hubpages, a top-to-bottom ordering of the news article references isincomplete because it does not account for the horizontal positioning ofthe references. In some implementations, the positions of the referencesin these hub pages are determined by analyzing their DOMs and stylesheets.

In some implementations, a hub page uses different formatting, which canbe defined in a style sheet, for news articles in different positions inthe hub page. For example, a style sheet for a hub page can specify thefollowing formatting: H1{text-align: center; font-size: 30pt} and H2{text-align: right; font-size: 20pt}. Some articles can have the H1formatting, and some other articles can have the H2 formatting. Theanalysis can include learning the different formatting employed by a hubpage for its article references and using that information to find thepositions of the references.

An example of analysis of a DOM structure of a hub page will now bedescribed in reference to FIGS. 5A and 5B. FIG. 5A illustrates anexample hub page 500 as displayed in a web browser. The hub page 500includes references to news articles 502 and 504. Below reference 502,there are references to news articles 506 and 508 that are related toarticle reference 502. Based on a visual inspection of the hub page 500,it can be determined that article reference 502 is the most prominent atthat time and that article reference 504 is the next most prominent.However, if the page is analyzed strictly from top to bottom, newsarticle references 506 and 508 can be mistaken to be more prominent thanarticle reference 504.

FIG. 5B illustrates a Document Object Model (DOM) graph of the hub pageillustrated in FIG. 5A. The DOM structure 520 resembles a tree structurewith nodes 524 and 526 as children of node 522. Nodes 528, 530, and 532,which correspond to article references 502, 506, and 508, respectively,are the children of node 524. Node 534, which corresponds to articlereference 534, is the child of node 526. Thus, the article references inthe hub page are in different branches of the DOM structure 520. Nodes528, 530, and 532 can be grouped into a Group A 536, and node 534 can beassigned to a Group B 538 in the DOM structure. Among the nodes in GroupA 536, node 528, which corresponds to the top most article reference 502within Group A, is included in the ordering of article references andthe other two article references 506 and 508, which correspond to nodes530 and 532, are ignored; the references 502, 506, and 508 are countedas one reference because their corresponding nodes have the same parentnode 524 in the DOM structure. Thus, for hub page 500, the ordering ofarticle references is: 502, 504, rather than 502, 506, 508, 504.

The news article references can be ordered based on the determinedpositions. For example, if the hub page is one where left-to-rightordering is the accepted convention (e.g., if the hub page language isone that is written from left to right), an ordering can be assigned tothe news article references in a top-to-bottom, left-to-right order. Inanother example, if the hub page is one where right-to-left ordering isthe accepted convention (e.g., if the hub page language is one that iswritten from right to left), an ordering can be assigned to the newsarticle references in a top-to-bottom, right-to-left order. As a furtherexample, if the hub page has a left, middle, and right section, anordering can be assigned to the news article references such thatreferences in the middle section, for the same vertical position, areordered higher than references in the left or right sections. The newsarticles are then assigned a score based on the determined order.

In some implementations, the analysis of the DOM and style sheet alsoincludes identifying the language of the hub page and applying differentreference ordering rules depending on the language. For example, if thehub page is written in a language that has a right-to-left orientation,the ordering rule can be different than the rule that is applied to ahub page written in a language with a left-to-right orientation.

In some implementations, the reference position is one of multiplecriteria for determining, more generally, a prominence score of a newsarticle in a hub page. The prominence of a news article reference in ahub page is used as a metric of the importance of the news article fromthe perspective of an editor or publisher of the hub page. Othercriteria of prominence can include whether the reference includes asnippet, summary, or synopsis of the corresponding news article; thefont size of the anchor text or a headline text of the referencehyperlink; formatting (e.g., bold, italics, etc.) of the anchor text orheadline text of the reference; whether there is an image associatedwith the reference; the size of any snippet; and total number ofreferences to the news article.

In some implementations, the prominence of a news article is combinedwith an importance measure (e.g., an importance score) that is based onclick logs of the news articles. A click log records selections by users(e.g., by mouse clicks) of the corresponding hyperlink to the newsarticle. An importance score for a hyperlink with respect to a hub pagecan be calculated using the number of selections of a hyperlink,weighted by the position of the hyperlink in the hub page. The weightingby position can counter users' bias in favor of hyperlinks that appearnearer the top of a hub page.

In some implementations, the prominence score for an article iscalculated as follows. Prominence scores are calculated only for the top10 references in the ordering in a hub page. The top-most reference isassigned a reference order of 1; the next reference has a referenceorder of 2, and so on, up to a reference order of 10. The prominencescore for an article is calculated using the following formula:prominence_score=reference_position_score+(optional)scores of othercomponentswhere reference_position_score is C1/power(reference_order, C2) asdescribed above. The scores of other components are a combination (e.g.,a linear combination) of scores determined based on other criteria(e.g., inclusion of a snippet, font size, formatting, etc.). If thereare no other score components than the reference position score, thenthe prominence score is simply C1/power(reference_order, C2).

The system determines scores for the news articles based on thedetermined prominence of the corresponding references and optionallyother components (306). In some implementations, the reference positionscore, or more generally a prominence score, is one component ofmultiple components that make up the score for a news article. Forexample, other components can include the quality of the news sourcehosting the news article, freshness of the news article (how recentlywas the news article published), length of the news article, and“novelty” of the subject matter of the news article.

In some implementations, the score of a news article is calculated as alinear combination of scores for multiple components. For example, anews article score S can be calculated using the formula S=αA+βB+δC+ . .. , where α, β, and δ are constant weights assigned to each component,and A, B, C are the scores for each component.

In some other implementations, the article score S can be the following:S=reference_position_score+scores of other componentswhere the reference_position_score is as described above and the scoresof other components are a combination (e.g., a linear combination) ofscores determined based on any criteria other than reference position.Thus, the scores of other components can includes scores determinedbased on whether the reference includes a snippet, reference font sizeand formatting, quality of the news source hosting the news article,freshness of the news article, length of the news article, and “novelty”of the subject matter of the news article.

An example of how to determine the “novelty” of the subject matter of anews article is disclosed in U.S. patent application Ser. No.11/378,628, titled “Detecting Novel Document Content,” filed Mar. 20,2006, which is incorporated by reference herein in its entirety.

In some implementations, article grouping module 216 groups newsarticles into news article groups. In some implementations, a newsarticle group is a cluster of news articles that report on, describe, orare related to the same news event or similar news events. News articlescan be clustered by first identifying keywords (or more generally,terms, which can include words, phrases, numbers, characters, and so on)within the respective news articles. For example, in someimplementations, keywords in a news article are identified bycalculating the term frequency-inverse document frequency (TF-IDF)values of words in the news article, and selecting those words whoseTF-IDF values exceed a threshold. A keyword vector is generated for eachof the news articles. Each vector includes the TF-IDF values of thekeywords in the respective news article. Similarity scores of newsarticles are then calculated by calculating the cosine similarities ofpairs of news articles, with the cosine similarity between two newsarticles being the dot product of the word vectors corresponding to thetwo news articles, divided by the product of the magnitudes of the twovectors. The news articles are then clustered into news article groupsbased on the cosine similarities using any convenient clusteringtechnique. One convenient clustering technique is agglomerativehierarchal clustering.

The article group scoring module 218 can determine scores for newsarticle groups. In some implementations, the score for a news articlegroup is a derived from the scores of the news articles clustered intothe news article group and optionally other components. An example of acombination of the scores of the news articles in a news article groupis an arithmetic mean or median of the news article scores. As anotherexample, the group score is calculated by adding up the scores of thenews articles in the group and then the sum of the scores is scaled down(e.g., by taking a square root of the sum). The optional othercomponents can include the rate at which the news articles clusteredunder the news article group are being published. The news articlegroup, and thus their respective stories, can be ranked based on thecalculated scores. Whenever a user submits a query for news articles,the search engine server 208 generates an output that presents, assearch results, news article groups that satisfy the query in theirranking order. The user can select a news article group to view alisting of news articles in that news article group. When the useraccesses a user interface page of the search engine server, the searchengine server 209 can generate, and transmit for presentation to theuser, the user interface page with one or more news article references,organized based on news article groups, where the groups for whichreferences are displayed are the highest scoring.

In some implementations, the articles in a group are re-scored afterbeing grouped into news article groups, before the news article groupscore is calculated. For example, the articles in the group can beordered within the group and scored based on the order within the group.

In some implementations, news articles and news article groups arescored and ranked within topical categories. For example, a businessnews article can be scored and ranked with respect to articles in thebusiness category and/or with respect to articles overall. The articlecan have an overall score and a score with respect to the businesscategory. A news article can be associated with a topical category ifthe corresponding reference is found in a hub page that is associatedwith the topical category (e.g., reference to a sports news articlefound in a sports section hub page) or if the corresponding reference isgrouped with other references associated with the topical category inthe hub page (e.g., business news article references are presentedtogether in the hub page under a “business” heading). Examples of newsarticle references include references 416 and 418 shown in FIG. 4B. Theanalysis of a hub page to identify news article references can includeidentifying any topical categories with which a news article referencecan be associated and identifying news article references associatedwith the identified categories. News articles corresponding toreferences that are associated with a category are scored with respectto the category.

In some implementations, stale hub pages, i.e., hub pages which haven'tbeen updated with new content within some period of time (e.g., last fewhours), are filtered. Hub pages can be crawled regularly and the contentof a hub page at one crawl is compared to the content of the hub page ata subsequent crawl (e.g., a crawl that occurred 5 hours since than thelast crawl). If the content is the same, then that hub page isdisregarded in the scoring and ranking process.

FIGS. 4A and 4B illustrate example hub pages as presented by anapplication such as a web browser. FIG. 4A shows a rendered hub page 400that includes news article references 402, 404. A news article referenceincludes a hyperlink to a news article document corresponding to thereference. In some implementations, a news article reference can alsoinclude a snippet or a synopsis of the news article (e.g., asillustrated by news article references 402-A, 402-B, 402-C) andoptionally an indication of a time of publication or posting of the newsarticle (e.g., a timestamp of the news article or a time elapsed sincepublication of the news article).

In some hub pages, the news article references are arranged from top tobottom (i.e., vertically) within the hub page 400, as illustrated inFIG. 4A. For such pages, for a referenced news article, the referenceposition score of the news article can be determined based on theposition of its corresponding reference in the hub page relative toother references in the hub page. News articles corresponding toreferences nearer to the top of the hub page are scored higher. Forexample, the news article corresponding to reference 402-A correspondsto the highest position within the hub page 400, and thus has thehighest-valued reference position score. The news article correspondingto the next highest positioned reference, reference 402-B, has the nexthighest-valued reference position score, and so on.

In some implementations, the difference between reference positionscores for one referenced news article to the subsequent referenced newsarticle is constant throughout the hub page. In such an implementation,the difference between the reference position scores for the newsarticles corresponding to references 402-A and 402-B could be 0.5; inwhich case the difference between the reference position scores for thenews articles corresponding to references 402-B and 402-C would be 0.5;and so on down the hub page.

In some other implementations, the difference between reference positionscores for consecutive referenced news articles can be different as onegoes down the hub page. For example, the reference position scores coulddecrease monotonically, by one constant value, for the first 5referenced news articles, monotonically by another constant value forthe 6th thru 10th referenced news articles, monotonically by yet anotherconstant value for the 11th thru 15th referenced news articles, and soon.

FIG. 4B illustrates a hub page 410 that includes news article referencesarranged horizontally and vertically within the rendered hub page 410.Hub page 410 includes news article references 412-A thru 412-C, 414,416, and 418. News article references 416 include references to newsarticles about business news. News article references 418 includereferences to news articles about sports news.

The hub page 410 can be analyzed to identify the news article referencesin the hub page, as described above in reference to FIG. 3. The newsarticles can be scored based on the positions of the correspondingreferences in the hub page. In some implementations, the referenceposition score for a news article is determined based on one or morecriteria that take into account both the vertical and horizontalposition of the corresponding reference. For example, the news articlecorresponding to reference 412-B may be given a higher referenceposition score than the news article corresponding to reference 412-Cdespite the same vertical positions of the references 412-B and 412-C.

Hub page 410 includes news article references 416 and 418 that aregrouped under respective topical categories, in regions of the hub pagededicated to the respective topical categories. Reference positionscores within a respective topical category can be determined for thenews articles corresponding to references 416 and 418. For example, thenews articles corresponding to the references 416 can be given areference position score for business news articles (because thereference is in the section for business news articles), as well as anoverall reference position score.

The disclosed and other embodiments and the functional operationsdescribed in this specification can be implemented in digital electroniccircuitry, or in computer software, firmware, or hardware, including thestructures disclosed in this specification and their structuralequivalents, or in combinations of one or more of them. The disclosedand other embodiments can be implemented as one or more computer programproducts, i.e., one or more modules of computer program instructionsencoded on a computer-readable medium for execution by, or to controlthe operation of, data processing apparatus. The computer-readablemedium can be a machine-readable storage device, a machine-readablestorage substrate, a memory device, a composition of matter effecting amachine-readable propagated signal, or a combination of one or morethem. The term “data processing apparatus” encompasses all apparatus,devices, and machines for processing data, including by way of example aprogrammable processor, a computer, or multiple processors or computers.The apparatus can include, in addition to hardware, code that creates anexecution environment for the computer program in question, e.g., codethat constitutes processor firmware, a protocol stack, a databasemanagement system, an operating system, or a combination of one or moreof them. A propagated signal is an artificially generated signal, e.g.,a machine-generated electrical, optical, or electromagnetic signal, thatis generated to encode information for transmission to suitable receiverapparatus.

A computer program (also known as a program, software, softwareapplication, script, or code) can be written in any form of programminglanguage, including compiled or interpreted languages, and it can bedeployed in any form, including as a stand-alone program or as a module,component, subroutine, or other unit suitable for use in a computingenvironment. A computer program does not necessarily correspond to afile in a file system. A program can be stored in a portion of a filethat holds other programs or data (e.g., one or more scripts stored in amarkup language document), in a single file dedicated to the program inquestion, or in multiple coordinated files (e.g., files that store oneor more modules, sub-programs, or portions of code). A computer programcan be deployed to be executed on one computer or on multiple computersthat are located at one site or distributed across multiple sites andinterconnected by a communication network.

The processes and logic flows described in this specification can beperformed by one or more programmable processors executing one or morecomputer programs to perform functions by operating on input data andgenerating output. The processes and logic flows can also be performedby, and apparatus can also be implemented as, special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) or an ASIC(application-specific integrated circuit).

Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processors of any kind of digital computer. Generally, aprocessor will receive instructions and data from a read-only memory ora random access memory or both. The essential elements of a computer area processor for performing instructions and one or more memory devicesfor storing instructions and data. Generally, a computer will alsoinclude, or be operatively coupled to receive data from or transfer datato, or both, one or more mass storage devices for storing data, e.g.,magnetic, magneto-optical disks, or optical disks. However, a computerneed not have such devices. Computer-readable media suitable for storingcomputer program instructions and data include all forms of non-volatilememory, media and memory devices, including by way of examplesemiconductor memory devices, e.g., EPROM, EEPROM, and flash memorydevices; magnetic disks, e.g., internal hard disks or removable disks;magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor andthe memory can be supplemented by, or incorporated in, special purposelogic circuitry.

To provide for interaction with a user, the disclosed embodiments can beimplemented on a computer having a display device, e.g., a CRT (cathoderay tube) or LCD (liquid crystal display) monitor, for displayinginformation to the user and a keyboard and a pointing device, e.g., amouse or a trackball, by which the user can provide input to thecomputer. Other kinds of devices can be used to provide for interactionwith a user as well; for example, feedback provided to the user can beany form of sensory feedback, e.g., visual feedback, auditory feedback,or tactile feedback; and input from the user can be received in anyform, including acoustic, speech, or tactile input.

The disclosed embodiments can be implemented in a computing system thatincludes a back-end component, e.g., as a data server, or that includesa middleware component, e.g., an application server, or that includes afront-end component, e.g., a client computer having a graphical userinterface or a web browser through which a user can interact with animplementation of what is disclosed here, or any combination of one ormore such back-end, middleware, or front-end components. The componentsof the system can be interconnected by any form or medium of digitaldata communication, e.g., a communication network. Examples ofcommunication networks include a local area network (“LAN”) and a widearea network (“WAN”), e.g., the Internet.

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other.

While this specification contains many specifics, these should not beconstrued as limitations on the scope of what being claims or of whatmay be claimed, but rather as descriptions of features specific toparticular embodiments. Certain features that are described in thisspecification in the context of separate embodiments can also beimplemented in combination in a single embodiment. Conversely, variousfeatures that are described in the context of a single embodiment canalso be implemented in multiple embodiments separately or in anysuitable subcombination. Moreover, although features may be describedabove as acting in certain combinations and even initially claimed assuch, one or more features from a claimed combination can in some casesbe excised from the combination, and the claimed combination may bedirected to a subcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understand as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processingmay be advantageous. Moreover, the separation of various systemcomponents in the embodiments described above should not be understoodas requiring such separation in all embodiments, and it should beunderstood that the described program components and systems cangenerally be integrated together in a single software product orpackaged into multiple software products.

Thus, particular embodiments have been described. Other embodiments arewithin the scope of the following claims.

What is claimed is:
 1. A method performed by one or more computers, themethod comprising: identifying a plurality of hub pages that eachinclude at least a threshold number of hyperlinks to online content;determining, for each of at least two of the hyperlinks and by one ormore computers, a respective reference position score based on relativepresentation positions of the hyperlinks in the hub pages, the relativepresentation position of each hyperlink in a hub page being a locationin the hub page at which the hyperlink is presented relative tolocations in the hub page at which different hyperlinks to differentonline content are presented; deriving, by one or more computers, scoresfor online content to which the hyperlinks link, the score for aparticular portion of online content being derived using the respectivereference position scores for the hyperlinks to the particular portionof online content; and providing data representing the scores.
 2. Themethod of claim 1, wherein determining, for each of at least two of thehyperlinks, a respective reference position score comprises selecting,for each of the at least two hyperlinks, a respective reference positionscore that is proportional to a measure of prominence of the relativepresentation position at which the hyperlink is presented on the hubpage.
 3. The method of claim 2, wherein determining the respectivereference position score further comprises determining a prominence ofthe relative presentation position at which the hyperlink is presentedon the hub page, the prominence of the relative presentation positionbeing based on an analysis of one or more of: an HTML source code of thehub page, a Document Object Model of the hub page, or a style sheet ofthe hub page.
 4. The method of claim 1, further comprising: grouping theonline content to which the hyperlinks link into one or more contentgroups, the content groups including a first content group; anddetermining a first group score for the first content group based on thecontent scores of the content grouped into the first content group. 5.The method of claim 4, further comprising presenting the content groupsand corresponding online content in response to a search query, whereinthe content groups are ordered based on their respective content groupscores.
 6. The method of claim 1, wherein: identifying a plurality ofhub pages comprises identifying hub pages that include hyperlinks toonline content that belongs to a same topical category; and determininga respective reference position score for each of the at least twohyperlinks comprises determining a respective reference position scorethat is based on the relative presentation position of the hyperlinkrelative to the different hyperlinks to the different online contentthat belongs to the same topical category.
 7. The method of claim 1,wherein deriving scores for the online content comprises: deriving thescore for each portion of online content to which the hyperlinks linkbased on one or more of: the relative presentation positions of thehyperlinks to the portion of online content on each of the hub pages, anamount of text presented with the hyperlinks to the portion of onlinecontent, font sizes of the text presented with the hyperlinks to theportion of online content, formatting of the text presented with thehyperlinks to the portion of online content, or presentation of an imagewith the hyperlinks.
 8. A system, comprising: one or more computers; anda computer storage device storing instructions that upon execution bythe one or more computers cause the one or more computers to performoperations comprising: identifying a plurality of hub pages that eachinclude at least a threshold number of hyperlinks to online content;determining, for each of at least two of the hyperlinks, a respectivereference position score based on relative presentation positions of thehyperlinks in the hub pages, the relative presentation position of eachhyperlink in a hub page being a location in the hub page at which thehyperlink is presented relative to locations in the hub page at whichdifferent hyperlinks to different online content are presented; derivingscores for online content to which the hyperlinks link, the score for aparticular portion of online content being derived using the respectivereference position scores for the hyperlinks to the particular portionof online content; and providing data representing the scores.
 9. Thesystem of claim 8, wherein determining, for each of at least two of thehyperlinks, a respective reference position score comprises selecting,for each of the at least two hyperlinks, a respective reference positionscore that is proportional to a measure of prominence of the relativepresentation position at which the hyperlink is presented on the hubpage.
 10. The system of claim 9, wherein determining the respectivereference position score further comprises determining a prominence ofthe relative presentation position at which the hyperlink is presentedon the hub page, the prominence of the relative presentation positionbeing based on an analysis of one or more of: an HTML source code of thehub page, a Document Object Model of the hub page, or a style sheet ofthe hub page.
 11. The system of claim 8, wherein the instructions causethe one or more computers to perform operations comprising: grouping theonline content to which the hyperlinks link into one or more contentgroups, the content groups including a first content group; anddetermining a first group score for the first content group based on thescores of the content grouped into the first content group.
 12. Thesystem of claim 11, wherein the instructions cause the one or morecomputers to perform operations including presenting the content groupsand corresponding online content in response to a search query, whereinthe content groups are ordered based on their respective content groupscores.
 13. The system of claim 8, wherein: identifying a plurality ofhub pages comprises identifying hub pages that include hyperlinks toonline content that belongs to a same topical category; and determininga respective reference position score for each of the at least twohyperlinks comprises determining a respective reference position scorethat is based on the relative presentation position of the hyperlinkrelative to the different hyperlinks to the different online contentthat belongs to the same topical category.
 14. The system of claim 8,wherein deriving scores for the online content comprises deriving thescore for each portion of online content to which the hyperlinks linkbased on one or more of: the relative presentation positions of thehyperlinks to the portion of online content on each of the hub pages, anamount of text presented with the hyperlinks to the portion of onlinecontent, font sizes of the text presented with the hyperlinks to theportion of online content, formatting of the text presented with thehyperlinks to the portion of online content, or presentation of an imagewith the hyperlinks.
 15. A non-transitory computer storage mediumencoded with a computer program, the program comprising instructionsthat when executed by one or more computers cause the one or morecomputers to perform operations comprising: identifying a plurality ofhub pages that each include at least a threshold number of hyperlinks toonline content; determining, for each of at least two of the hyperlinks,a respective reference position score based on relative presentationpositions of the hyperlinks in the hub pages, the relative presentationposition of each hyperlink in a hub page being a location in the hubpage at which the hyperlink is presented relative to locations in thehub page at which other hyperlinks to other content are presented;deriving scores for online content to which the hyperlinks link, thescore for a particular portion of online content being derived using therespective reference position scores for the hyperlinks to theparticular portion of online content; and providing data representingthe scores.
 16. The computer storage medium of claim 15, whereindetermining, for at least two of the hyperlinks, a respective referenceposition score comprises selecting, for each of the at least twohyperlinks, a respective reference position score that is proportionalto a measure of prominence of the relative presentation position atwhich the hyperlink is presented on the hub page.
 17. The computerstorage medium of claim 16, wherein determining the respective referenceposition score further comprises determining a prominence of therelative presentation position at which the hyperlink is presented onthe hub page, the prominence of the relative presentation position beingbased on an analysis of one or more of: an HTML source code of the hubpage, a Document Object Model of the hub page, or a style sheet of thehub page.
 18. The computer storage medium of claim 15, wherein theinstructions cause the one or more computers to perform operationscomprising: grouping the online content to which the hyperlinks linkinto one or more content groups, the content groups including a firstcontent group; and determining a first group score for the first contentgroup based on the scores of the content grouped into the first contentgroup.
 19. The computer storage medium of claim 18, wherein theinstructions cause the one or more computers to perform operationsincluding presenting the content groups and corresponding online contentin response to a search query, wherein the content groups are orderedbased on their respective content group scores.
 20. The computer storagemedium of claim 15, wherein deriving scores for the online contentcomprises deriving the score for each portion of online content to whichthe hyperlinks link based on one or more of: the relative presentationpositions of the hyperlinks to the portion of online content on each ofthe hub pages, an amount of text presented with the hyperlinks to theportion of online content, font sizes of the text presented with thehyperlinks to the portion of online content, formatting of the textpresented with the hyperlinks to the portion of online content, orpresentation of an image with the hyperlinks.